Recent Advances in Stochastic Gradient Descent in Deep Learning

In the age of artificial intelligence, best approach to handling huge amounts data is a tremendously motivating and hard problem. Among machine learning models, stochastic gradient descent (SGD) not only simple but also very effective. This study provides detailed analysis contemporary state-of-the-art deep applications, such as natural language processing (NLP), visual processing, voice audio processing. Following that, this introduces several versions SGD its variant, which are already in PyTorch optimizer, including SGD, Adagrad, adadelta, RMSprop, Adam, AdamW, so on. Finally, we propose theoretical conditions under these methods applicable discover that there still gap between algorithms converge practical how bridge question for future.